Method and apparatus for determining information about a drug-containing vessel

ABSTRACT

Information about a drug-containing vessel is determined by capturing image data of the curved surface of a cylindrical portion of a drug-containing vessel. The image data is unfurled from around the curved surface, binarised, and a template matching algorithm employed to determine that the label information comprises candidate information about the vessel and/or the drug.

FIELD

This disclosure relates to the determination of information about adrug-containing vessel. In particular, but without limitation, thisdisclosure relates to a method and apparatus for determining informationabout a drug-containing vessel (primary pack) such as a syringe orcartridge that is contained within a medical device such as anautoinjector.

BACKGROUND

Patients that suffer from one or more of a variety of medical conditionssuch as multiple sclerosis, arthritis, growth hormone deficiency, TurnerSyndrome, and chronic renal failure may require regular percutaneousadministration of one or more medicaments. Although such administrationmay be performed by health professionals, in some cases administrationmay be performed by the patient themselves or their carer. Some medicaldevices, such as autoinjectors, are operable to receive adrug-containing vessel, such as a syringe or cartridge, and, uponactuation, percutaneously administer the drug to the patient.

SUMMARY

Aspects and features of the present disclosure are set out in theappended claims

BRIEF DESCRIPTION OF THE DRAWINGS

Examples of the present disclosure will now be explained with referenceto the accompanying drawings in which:

FIG. 1 shows a medical device;

FIG. 2 shows an example drug-containing vessel;

FIG. 3 shows alternative drug-containing vessel;

FIG. 4 shows a medical device carrying a drug-containing vessel;

FIG. 5 shows an example set up for whereby a single imaging device maybe used in conjunction with a pair of mirrors;

FIGS. 6 and 7 show an alternative arrangement using a “hall of mirrors”principal;

FIG. 8 shows an end on view of a drug-containing vessel and mirrorsetup;

FIG. 9 shows a perspective view of the setup of FIG. 8 ;

FIG. 10 shows a hybrid approach wherein a plurality of imaging devicesare arranged both to image directly a drug-containing vessel and also toimage respective mirrors;

FIG. 11 shows an example label for a drug-containing vessel;

FIG. 12 shows a pair of example perspective views of a rectangular labelthat has been wrapped around the curved surface of a cylindrical portionof a drug-containing vessel;

FIG. 13 illustrates how x and y pixel coordinates of a rectangular labeltranslate into imaging device coordinates;

FIG. 14 shows a number of reconstructed label patches;

FIG. 15 shows four reconstructed image patches;

FIG. 16 shows four example masks corresponding to the mappings thatrespectively produced the reconstructed patches of FIG. 15 ;

FIG. 17 shows three unfurled images following acquisition using theapparatus of FIG. 10 ;

FIG. 18 shows an unfurled image on the left hand side and a binarisedversion of the unfurled image on the right hand side;

FIG. 19 shows a large template along with three sub-tiles thereof;

FIG. 20 shows an example unfurled image alongside a response for astandard template matching algorithm;

FIG. 21 shows another example unfurled image alongside a response for astandard template matching algorithm; and

FIG. 22 illustrates a classification approach.

DETAILED DESCRIPTION

Once a drug-containing vessel has been loaded into a medical device,such as an auto injector, it is to be expected that the user willattempt to actuate the device so as to administer the drug. However, ifthe wrong drug-containing vessel is loaded into the medical device, thenthe patient may not receive the drug that they need, may receive thewrong dosage of the drug, or may receive a drug that it is notappropriate for them to receive—all of which can be seriouslydeleterious to the patient. Checking whether a drug-containing vessel isthe correct one for a patient is generally done by visually inspecting alabel borne on the curved surface of a cylindrical portion of thedrug-containing vessel. Although drug-containing vessels could bemechanically configured, for example by way of keying, so as to makethem recognisable from their shape, and/or could be provided with otheridentification means—such as RFID tags, the fact that differentmanufacturers produce drugs and provide them in differently shapedcontainers means that visual inspection of the containers' labels iscurrently the best approach for verifying the contents of adrug-containing vessel.

In an alternative embodiment, after use the patient will discard thedrug-containing vessel. Discarding of the drug containing vessel may becarried out using a vessel configured to receive such discarded emptydrug vessel. In order to improve compliance monitoring the discardvessel may need to recognize the empty drug-containing vessel. So in analternative embodiment the invention is directed to a device and methodfor recognizing and identifying the drug-containing vessel such as to beable to record the drug containing vessel that is disposed of and itsidentity.

There is described herein an approach for determining information abouta drug-containing vessel that is carried by the curved surface of acylindrical portion of a drug-containing vessel so that a drug-agnosticmedical device, such as an auto injector, can determine which drug hasbeen loaded into it and can hence determine whether or not to enableadministration of the drug.

FIG. 1 shows a medical device 110 a vessel holder 112 arranged to hold acontaining vessel such as a syringe or a cartridge. A drug deliverymechanism 114 is operable to act with a drug-containing vessel held bythe vessel holder 112 so as to administer the drug contained in thatvessel when the medical device is adjacent to the patient (not shown)and the drug delivery mechanism 114 is instructed to do so by aprocessor 116. The drug delivery mechanism being operable to move atleast a part of the vessel holder 112 along with the drug-containingvessel and a needle coupled thereto along a path from within theinterior of the medical device 110 to the exterior of the medical device110 so as to puncture a patient's dermis. The drug delivery mechanism114 is further operable to depress a plunger of the drug-containingvessel so as to cause the drug contained thereby to be expelled from thedrug-containing vessel and via the hypodermic needle into the patient'stissue. The medical device 110 further comprises a label imager 118operable to image a label borne on a curved surface of a cylindricalportion of the drug-containing vessel and to provide image dataconsequent to that imaging to the processor 116. The processor 116 isarranged to operate in accordance with instructions stored in memory 120so as to receive images from the label imager 118 and to control thedrug delivery mechanism 114. Processor 116 may further be coupled to aninput/output component 122 by which the processor 116 may receiveinstructions for example template information with regard to drugs anddosages associated with the medical device and/or may output alarms, forexample to indicate that an unexpected drug-containing vessel has beeninserted into the medical device 110. FIG. 2 shows an exampledrug-containing vessel 210 which is a syringe having: a hypodermicneedle 212, a cylindrical portion 214, a plunger 216, finger guards 218,and an actuation end 220. The cylindrical portion 214 of the syringe 210bears a label that was rectangular but has been wrapped around thecurved surface of the cylinder. The label 222 contains information aboutthe drug contained by the syringe 210 and further contains informationabout the amount of drug contained by the syringe—in this case “5 mg”.FIG. 3 shows alternative drug-containing vessel 310—in this case acartridge having a plunger 312 operable to slide within in a cylindricalportion of the cartridge 314 so that, when a needle pierces a pierceablemembrane 316 and the plunger is moved in direction A, the drug isexpelled via the needle.

A number of different approaches for imaging labels borne by the curvedsurfaces of the vessel the cylindrical portions of drug-containingvessels have been contemplated and will now be described.

Scanning Imaging Devices

FIG. 4 shows a medical device 410 carrying a drug-containing vessel (inthis case a syringe) 412. The medical device 410 has a pair of imagingdevices 414, 416 operable to move along respective rails 418, 420 uponactuation of respective actuators 422, 424 in direction B-C or C-B so asto enable the imaging devices 414, 416 to capture a plurality of imagesof opposite sides of the curved surface of the cylindrical portion ofthe drug-containing vessel 412. Imaging devices 414, 416 form part ofthe label imager 118 and consequently relay the acquired images as imagedata to the processor 116.

Once received by the processor 116, images from each imaging device thatwere acquired at different time points as that imaging device movedalong its rail may be fused or blended so as to produce a single imagefrom each imaging device for subsequent processing.

Although the example of FIG. 4 uses a pair of imaging devices mounted onrails, approaches are also contemplated wherein the number of imagingdevices is greater than two, for example any of three to ten or evenmore. Furthermore, the imaging devices need not be mounted on rails andalternative mechanisms could be employed in order to enable them to scanthe cylindrical portion of the drug-containing vessel 412.

Hall of Mirrors

FIG. 5 shows an example set up whereby a single imaging device 510 maybe used in conjunction with a pair of mirrors 512, 514 to image adrug-containing vessel 516 using the “hall of mirrors” approach.Advantageously this approach avoids the imaging device needing to benear the end of the needle and so does not interfere with moving theneedle towards a patient's skin or actuation of the plunger of thedrug-containing vessel. However for some drug-containing vessels (suchas those having syringe flanges) portions projecting from thedrug-containing vessel can obscure parts of the label. FIGS. 6 and 7show an alternative arrangement using a “hall of mirrors” principalwhereby an imaging device 610 is positioned near the tip of a needle ofa drug-containing vessel 612 that is positioned adjacent to a pair ofmirrors 614, 616 whose planes are angled with respect to one another. InFIG. 6 , reflections of the drug-containing vessel 612 can clearly beseen at positions 618 and 620.

FIG. 8 shows an end on view of a drug-containing vessel 810 (in thiscase a cartridge) that lies within a hole 812 of a block 814 that isarranged to translate over the drug-containing vessel 810 in directionD-E and E-D of FIG. 9 . The block 814 further comprises a plurality ofmirrors 816 angled away from the long axis of the drug-containing vessel810 so that, when the block 814 is translated in direction D-E or E-Dand an imaging device (not shown in either of FIG. 8 or 9 ), images amirror that is placed at an angle (in this case 45°) with respect to thedirection D-E, the curved surface of the cylindrical portion of thedrug-containing vessel 810 is scanned and imaged. As the mirror 818 liesalong the path that a needle connected to the drug-containing vessel 810would traverse in order to exit the medical device, an elliptical hole820 is provided in the mirror 818.

Hybrid Approach

FIG. 10 shows a hybrid approach wherein a plurality of imaging devices1010, 1012 are arranged both to image directly a drug-containing vessel1014 and also to image respective mirrors 1016, 1018 that are positionedso that each respective imaging device sees a reflective portion of thecurved surface of the drug-containing vessel 1014. In this instance theimaging devices 1010, 1012 are arranged diametrically opposite oneanother with respect to a long axis of the drug-containing vessel 1014and the mirrors 1016, 1018 are arranged diametrically opposite oneanother with respect to the long axis of the drug-containing vessel 1014and at an angle (in this case 45°) so that each imaging device is ableto acquire images of the drug-containing vessel from two differentperspectives.

Image Unfurling

Labels that are applied to drug-containing vessels can either beadhesive labels that are printed on before being furled around thecurved surface of the cylindrical portion of a drug-containing vessel ormay be printed or otherwise placed on the curved surface of thecylindrical portion of a drug-containing vessel. FIG. 11 shows anexample label 1110 for a drug-containing vessel containing informationabout the drug 1112, in this case that the drug is “saizen 5.83 mg/ml”and the label 1110 further contains information 1114 about thedrug-containing vessel, in particular that it is a “6 mg cartridge”.FIG. 11 further shows label 1110 when it has been furled around a curvedsurface corresponding to the curved surface of a cylindrical portion ofa drug-containing vessel 1116. When imaged, such a furled label willhave portions of the label information that it containsforeshortened—for example the words “somatropin 118” are foreshortenedin the furled label 1116 of FIG. 11 to the point where they aredifficult to discern. Accordingly, in order for an image of adrug-containing vessel to have label information extracted therefromwhilst avoiding or reducing the effects of label informationforeshortening, a relationship between the original rectangular label(which may be a hypothetical original rectangular label in the eventthat the label information was directly printed onto the drug-containingvessel and the data received by an imaging device imaging thedrug-containing vessel needs to be established. The matter is somewhatconfounded by the fact that, due to the label have being furled about acylinder, a projection on the acquisition point of an imaging devicefrom the curved surface of the cylindrical portion of thedrug-containing vessel will often coincide with two points on thelabel—one from a near side to the imaging device and one from a far sideto the imaging device.

FIG. 12 shows a pair of example perspective views of a rectangular labelthat has been wrapped around the curved surface of a cylindrical portionof a drug-containing vessel. On the left of FIG. 12 , the twoperspective views 1210, 1212 illustrate that respective portions of thelabel are not visible due to the other portions of the label being infront of them. Accordingly, when the information from an imaging devicethat acquires such a perspective image is unfurled so as to set it outas a rectangular label 1214, 1216 not all of the label can bereconstructed. Accordingly, one approach, which may be optional, forunwrapping a label from the curved surface of the cylindrical portion ofa drug-containing vessel includes removing mappings that would beobscured from the imaging device by other portions of the label that liein front thereof. However, as drug-containing vessels and the drugscontained therein are sometimes somewhat transparent, a further approachmay instead keep that information and look to use it to help read thelabel.

FIG. 13 illustrates how x and y pixel coordinates of a rectangular labeltranslate into photo (or imaging device) coordinates. In particular,taking at step s1310 a set of x, y pixel labels of the rectangularlabel, and taking into account from step s1312 the expected cylinderradius and cylinder length of the cylindrical portion of thedrug-containing vessel, at step s1314, the pixel labels can be wrappedaround the cylinder so as to project them into 3D world coordinates.Following a calibration step s1316 to calibrate the imaging device andthe device within which both the imaging device and the vessel holderare positioned, at step s1318, the 3D world coordinates can be projectedonto a 2D imaging device plane before, at step s1320, mappings toportions of the curved surface of the cylinder that would be obscuredfrom the view of the imaging device can be removed so as to produce asan output at step s1322 a locations in the photo that correspond to thex,y pixel positions in the label image. Once it is known where any pixelin the rectangular label will map onto in an image of the cylinder, eachpixel in the rectangular label can be sampled in the imaging deviceimage in order unfurl the imaging device image. As the locations in theimaging device image that are to sampled will not necessarily coincidewith integer pixel locations, interpolation approaches, such as nearestneighbour, bilinear, and/or higher order approaches such as b-splineinterpolation, can be used in order to interpolate the imaging deviceimage.

As an example, the size of the label is obtained from the cylinderlength and radius parameters: height=2*pi*radius, width=length. A sizeof the image in pixels is obtained by scaling this considering thedesired resolution of the recreated label in DPI (dots per inch). Thelabel is then wrapped around a cylinder in a 3D world. The origin of theworld coordinate system is chosen such that it coincides with the centreof the circle in the base of the cylinder. Axes x and y are then in theplane containing this circle, while the z axis is along the length ofthe cylinder. The x axis can point to where the label is “glued”, whichcorresponds to the bottom line of the label 1116. From this lineupwards, each line of the label is placed on the circle at an angleincreasing anticlockwise. The wrapping, or mapping from (x,y) of thelabel in pixels to (x,y,z) of the cylinder in millimetres is performedas:

${x_{cylinder} = {r_{cylinder}{\cos\left( {\frac{{y\max} - y}{y\max} \times 2\pi} \right)}}}{y_{cylinder} = {r_{cylinder}{\sin\left( {\frac{{y\max} - y}{y\max} \times 2\pi} \right)}}}{z_{cylinder} = {\frac{{x\max} - x}{{labelwidth}_{px}} \times {labelwidth}_{mm}}}$

This operation is carried out for each (x,y) pixel for the given labelsize and gives a physical location (in mm) in the world for each pixel.

Given a cylinder existing in the world, it is desired to estimate whereeach of its (x,y,z) voxels would appear when a photo of it is taken froma known position, with an imaging device of known parameters. FIG. 11also shows the image coordinate system and its origin in the top leftcorner of the image. The mapping from world to photo pixels can be foundas:

${\begin{bmatrix}u \\v \\w \\1\end{bmatrix} = {P\begin{bmatrix}x_{cyl} \\y_{cyl} \\z_{cyl} \\1\end{bmatrix}}},{x_{photo} = \frac{u}{w}},{y_{photo} = \frac{v}{w}}$

Here P is a 4×4 homogenous imaging device transformation matrix:

$P = {K\begin{bmatrix} & R & & T \\0 & 0 & 0 & 1\end{bmatrix}}$

Intrinsic imaging device Parameters (K matrix): K is the matrixcontaining the intrinsic imaging device parameters, and depends on theimaging device focal length, on the sensor size and on the position ofthe optical centre. It is essentially a scaling and translation matrixthat brings millimetre coordinates to pixel coordinates and accounts forthe optical centre (centre of image) not corresponding to the photoorigin (which is in the top left corner). The parameters depend on theimaging device in use, and a person skilled in the art have be wellacquainted with the use of calibration objections (such ascheckerboards) in order to determine focal lengths f_(x) and f_(y) inorder to determine K. In this case, K is:

$K = \begin{bmatrix}\frac{f_{x}}{s_{x}} & 0 & 0 & O_{x} \\0 & \frac{f_{y}}{s_{y}} & 0 & {Oy} \\0 & 0 & 1 & 0 \\0 & 0 & 0 & 1\end{bmatrix}$

Where s_(x) and s, are pixel sizes in millimetres, and O_(x) and O_(y),are the pixel coordinates of the optical centre in the image (whichshould be around the actual centre of the image, but does not coincidewith it).

Lens distortion parameters may also be obtained from the imaging devicecalibration process and used to adjust K accordingly.

Extrinsic Imaging Device Parameters (R and I matrices): as the worldcoordinate system does not coincide with the imaging device coordinatesystem (located in the optical centre, with axes as pictured in FIG. 11), the transformation expressed by rotation matrix R and translationvector T is required, such that the world points are expressed withrespect to the imaging device axes. A more intuitive manner ofspecifying the imaging device location can been employed that istailored to the image coordinate system. The parameters to be specifiedto obtain the R and T matrices are:

-   -   the physical location of the imaging device in world coordinates        (eye vector);    -   the coordinates of the point at which the imaging device is        looking (centre vector);    -   a direction in the world which, when projected on the image        plane, would point up, e.g. the world y axis (up vector).        Considering eye, centre, and up as row vectors, the R matrix is        obtained through the following steps:

${{\overset{\rightarrow}{L} = {\overset{\rightarrow}{centre} - \overset{\rightarrow}{eye}}};{\overset{\rightarrow}{L_{N}} = \frac{\overset{\rightarrow}{L}}{\overset{\rightarrow}{L}}};}{{\overset{\rightarrow}{s} = {\overset{\rightarrow}{L_{N}} \times \overset{\rightarrow}{up}}};{\overset{\rightarrow}{s_{N}} = \frac{\overset{\rightarrow}{s}}{\overset{\rightarrow}{s}}};}{{\overset{\rightarrow}{{up}^{\prime}} = {{- \overset{\rightarrow}{s_{N}}} \times \overset{\rightarrow}{L_{N}}}};}{{R = \begin{bmatrix}\overset{\rightarrow}{s_{N}} \\\overset{\rightarrow}{{up}^{\prime}} \\\overset{\rightarrow}{L_{N}}\end{bmatrix}};}$and the imaging device translation with respect to the newly rotatedcoordinate system is:T=−R*{right arrow over (eye)} ^(T)

This can also be understood by first applying a translation described by{right arrow over (eye)} followed by the rotation described by R.

Removing mappings to the hidden part of the cylinder: in a single imageand assuming that the label is opaque, only the portion of the labelthat lies on the nearside of the label to the imaging device is visibleto the imaging device as the portion of the label that lies on the farside of the cylinder will be obscured by the nearside portion. It istherefore desirable to identify and remove those pixels from the labelspace that are not visible in the imaged label space. To do this, it canbe assumed that, when viewing a cylindrical container, its long edges asthey appear in the image define a plane that bisects the cylinder,dividing it into a visible section and an obscured section. Accordingly,this divides the label space into the part of the label that is visibleand the part of the label that is obscure, as shown in FIG. 12 . Thelong edges of the cylinder may be identified among the points of themapping in the imaged label space using simple edge detection and, sincethe distance between the imaging device and the imaged label is knownfor each point in the imaged label space, the label pixels that map tothe far half of the container can be discarded. For set ups wheremirrors are employed so that images received by imaging devices containnot only directly imaged representations of the drug-containing vessel,but also reflected images of the drug-containing vessel, the mappingsare customised. For example, in the set-up of FIG. 10 , each imagingdevice 1010, 1012 will acquire an image with a direct representation ofthe drug-containing vessel 1014 and also a reflected image of thedrug-containing vessel 1014 that has been reflected from the respectivemirrors 1016, 1018. Accordingly, as each image captured by the imagingdevice contains two representations of the drug-containing vessel 1014,two different mappings may be applied to each image so as to extract theinformation associated with each view of the drug-containing vessel1014. Once information from the various views has been extracted, andunfurled then the various unfurled images may be patched together.

Continuing with the example set up of FIG. 10 , FIG. 14 shows, on theleft hand side, imaging devices 1010 and 1012 directly viewing thedrug-containing vessel 1014. On the right hand side of FIG. 14 , theangle of the reflecting mirrors 1016 and 1018 has been shown in anexaggerated manner to emphasise that each of the imaging devices 1010and 1012 will, in addition to directly imaging the drug-containingvessel 1014, also image reflected images 1410, 1412 of thedrug-containing vessel 1014. The parameters used to describe thisparticular setup are:

-   -   C1eye, C2eye: the (x,y,z) coordinates of camera 1 and camera 2        (expressed in the world coordinate system);    -   C1centre, C2centre: the (x,y,z) coordinates of the point at        which the camera is looking (the optical centre). This are not        unique, any point along the dotted line is suitable;    -   C1up, C2up: vectors used to compute the “up” direction of each        camera; (0,1,0) if the camera is “right side up” and (0,−1,0) if        the camera is upside down;    -   The vertical positions of the mirrors, h_(M1) and h_(M2) (h_(M2)        to be, considered negative);    -   The angles of the mirrors with a horizontal plane, α_(M1) and        α_(M2).

These parameters, are measured on the apparatus and expressed in theworld coordinate system. Measurements on the z axis may be lessinfluential on accurate label reading and so it may be that a fixedlength of cylinder is assumed, for example 40 mm, and it may be furtherassumed that the imaging devices are pointing towards the middle of it.

The setup of FIG. 10 provides four different pictures of the syringefrom different angles. Four mappings are, therefore, required:

1. From label to image 1: Cylinder in the world coordinates, viewed byimaging device 1010;

2. From label to image 1: Cylinder in the mirror 1 coordinate system,viewed by imaging device 1010;

3. From label to image 2: Cylinder in the world coordinates, viewed byimaging device 1012;

4. From label to image 2: Cylinder in the mirror 2 coordinate system,viewed by imaging device 1012;

Mappings 1 and 3 can be accounted for by the model as previouslydescribed, by using different projection matrices for each of theimaging devices that are looking at the drug-containing vessel 1014.Mappings for the mirror images are performed by expressing thecoordinates of the mirrored images in the same coordinate system as theoriginal cylinder (considered the “world” coordinate system, todistinguish it from the imaging devices coordinate systems). This can beimplemented in the same way as in above, but with an additional step ofmultiplying the vector of cylinder coordinates by a transformationmatrix that aligns the mirror coordinate system with the worldcoordinate system. This transformation matrix will be denoted by M, andcan be expressed as a function of the two mirror parameters, mirrorangle and mirror vertical position; it resembles a homogenous rotationand translation matrix, but would not be considered a proper rotationmatrix, as the mirror coordinate system does not follow the right handrule anymore. Revising the mapping model to accommodate mirror imagesgives:

${{\begin{bmatrix}u \\v \\w \\1\end{bmatrix} = {{PM}\begin{bmatrix}x_{cyl} \\y_{cyl} \\z_{cyl} \\1\end{bmatrix}}},{x_{photo} = \frac{u}{w}},{y_{photo} = \frac{v}{w}}}{where}{M = \begin{bmatrix}{\cos\left( {2\alpha} \right)} & {\sin\left( {2\alpha} \right)} & 0 & {{- h}{\sin\left( {2\alpha} \right)}} \\{\sin\left( {2\alpha} \right)} & {- {\cos\left( {2\alpha} \right)}} & 0 & {h + {h{\cos\left( {2\alpha} \right)}}} \\0 & 0 & 1 & 0 \\0 & 0 & 0 & 1\end{bmatrix}}$

The mappings from the imaging device image to the label can be generatedoffline and encompass the image distortions that occur. A mapping is apair of coordinates for each pixel (x,y) of the label: map_(x)(x,y),which gives the x coordinate of the corresponding pixel in the photo,and map_(y)(x,y), which gives the y coordinate. In other words,remapping performs the following assignment:label(x,y)=photo(map_(x)(x,y),map_(y)(x,y))

For the example of FIG. 10 , two of the four mappings are applied to oneof the two imaging device images and the other two mappings are appliedto the other one of the two images. This results in four reconstructedpatches of the label as shown in FIG. 15 which has a 3×2 matrix ofimages in which the first column shows the first and second imagingdevice images, the second column first row shows the reconstructed patchfor the mirror reflection of the drug-containing vessel from the firstimaging device image, the third column first row shows the reconstructedpatch for the directly viewed drug-containing vessel from the firstimaging device image, the second column second row shows thereconstructed patch for the mirror reflection of the drug-containingvessel from the second imaging device image, and the third column secondrow shows the reconstructed patch for the directly vieweddrug-containing vessel from the second imaging device image. In exampleswhere different numbers of imaging devices and/or mirrors are employedthen the number of mappings will vary accordingly and so will the numberof reconstructed patch images: for the apparatus of FIG. 4 thedrug-containing vessel is viewed directly by two imaging devices and sotwo reconstructed patch images may be created; for the apparatus of FIG.6 the drug-containing vessel is viewed directly once and two mirrorimages are viewed (all by the same imaging device) and so threereconstructed patch images may be created; for the apparatus of FIG. 9 asingle imaging device captures four reflected images and so fourreconstructed patch images may be created.

In cases where multiple reconstructed patch images are created, they maybe combined to form a single unfurled image of the label. Since eachpatch is at its correct location relative to the label, the problem ofreassembly comes down to blending the different patches together.Blending is preferable since sometimes the same part of the labelappears in multiple images, and due to this the reconstructed patchimages may overlap. As one possibility, each patch is multiplied by amask before adding it to the unfurled image of the label. Four examplemasks corresponding to the mappings that respectively produced thereconstructed patches of FIG. 15 is shown in FIG. 16 where whitecorresponds to 1 and black to 0. A person skilled in the art willrecognize that different blending techniques may be employed and will beacquainted with suitable alternative blending techniques. FIG. 17 showsthree unfurled images following acquisition using the apparatus of FIG.10 and the above described processing.

In situations where the diameter of the drug-containing vessel is notknown but can take a number of distinct specific potential values (i.e.a syringe of diameter 8 mm or 11 mm is expected), a mapping for eachpotential value can be used to create multiple unfurled images uponwhich subsequent processing can be performed. Such an approach avoidsthe need for dedicated processing to identify the size of thedrug-containing vessel and instead performs the processing subsequentlydescribed herein on each unfurled image before taking the bestindividual label determination result as being indicative of thecontents of the label.

Label Classification

The aim of the label classification step is to take each unfurled imageand produce a decision about whether or not the label is of apre-specified drug and/or dose.

One approach is to employ a template matching algorithm that searchesfor one or more given templates within the unfurled image. As anexample, one of these templates can be the name of the drug and otherswill be used that correspond to the dose and other important ordistinguishing features of the label.

Prior to performing the template matching, preprocessing approaches areapplied to the unfurled images so as to improve classificationperformance by removing irrelevant information while preserving relevantinformation. These pre-processing steps make the template matching morerobust and less computationally expensive.

For many labels, the prime requirement for classification is to matchthe shape of the corresponding template to that label and ideally otherfactors would be ignored. As an example, lighting can cause considerablevariations in the unfurled image. Accordingly, the pixel values of theunfurled images are thresholded to produce a binary image that removessuch variation and returns a much simpler two-level image which is stillclassifiable. An example of such a binarised image is given in FIG. 18which shows an unfurled image on the left hand side and a binarisedversion of the unfurled image on the right hand side.

Two approaches for performing binarisation will now be described. It maybe that one or other of the approaches is more appropriate for aspecific type of template. Where a drug or dosage is identified throughthe search for more than one template, then it may be necessary tocalculate multiple (differently) binarised versions of the unfurledimage so that the appropriate version is available for each templatesearch.

In the below equations the following notation is employed: R, G and Bare used to represent the red, green and blue values of a pixel. (x,y)is used to specify the specific location of a pixel in question. Forexample R(x,y) represents the red value of the pixel which is x pixelsin from the left hand edge and y pixels down from the top. And F is usedto represent the value of a pixel in the binarised image and is arrangedso that F will always have a value of 0 or 1.

The first binarisation approach may be suitable for templates wherepixels containing the text can be easily separated from the other pixelson the basis of intensity and involves first converting a colourunfurled image to greyscale. The greyscale value is calculated as aweighted sum of the RGB values:I=0.2989R+0.5780G+0.1140B

The greyscale image is binarised by applying an adaptive thresholdingalgorithm although a skilled person will recognize other thresholdingapproaches that could equally be employed including, but not limited to,the use of a global threshold. For each pixel in the now greyscaleunfurled image, the mean pixel value in a rectangular neighbourhood iscalculated and subtracted from the pixel in question. A fixed thresholdis then applied to the resulting image. This helps the thresholding tobe robust to variations in lighting across the image. The local meanintensity for each pixel in the image is calculated by:

${M\left( {x,y} \right)} = {\frac{1}{\left( {{2N} + 1} \right)^{2}}{\sum\limits_{u = {- N}}^{N}{\sum\limits_{v = {- N}}^{N}{I\left( {{x - u},{y - v}} \right)}}}}$The binary value for each pixel is then set to:

${F\left( {x,y} \right)} = \left\{ \begin{matrix}0 & {{{{if}{I\left( {x,y} \right)}} - {M\left( {x,y} \right)}} < C} \\1 & {{{{if}{I\left( {x,y} \right)}} - {M\left( {x,y} \right)}} \geq C}\end{matrix} \right.$

The second binarisation approach may be suitable for templates where itis important to use colour information to distinguish which pixelsbelong to the text and which to the background. In such cases theunfurled image is, where needed, converted from RBG to HSV(HueSaturation-Value). The advantage of this representation is that thecolour information is mostly contained in just the H value and this isrelatively robust to varying degrees of lighting. The HSV representationof the unfurled image is then binarised by selecting the pixels whose H,S and V values lie within a given range, which is centred on the colourof the text:F(x,y)=1 if T _(min) ^(H) ≤H(x,y)≤T _(max) ^(H) and T _(min) ^(S)≤S(x,y)≤T _(max) ^(S) and T _(min) ^(V) ≤V(x,y)≤T _(max) ^(V)

Once the unfurled image has been binarised, a template matching approachis employed which slides a template around the binarised unfurled imageand finds the point or points where the template best matches thebinarised unfurled image by evaluating a similarity measure between thetemplate and a number of candidate points in the binarised unfurledimage. This can be considered an optimization process wherein potentialtemplate locations in the unfurled image are evaluated in order todetermine a similarity score and the template location at which thesimilarity score is optimal (maximum or minimum depending on thesimilarity score) is searched for. Example optimisation approaches thatcould be employed would be to evaluate all possible template positionsor to use a gradient descent approach; other optimization approachescould also be employed.

Where there may be an issue with varying colour and intensity of acolour unfurled image, binarisation of the unfurled image may besuitable. As another possibility, template matching on colour images maybe used along with a suitable colour-employing similarity measure.

Template matching can be very robust to noise and is also tolerant tothe image being slightly out of focus (unlike edge or corner detectorswhich can require sharp edges); accordingly, the choice of a templatematching approach is sympathetic to the nature of the unfurled images.However, standard template matching is not so tolerant to rotations,scale factors, perspective distortions, and occlusions/missing parts ofan image.

Non-template-based shape-matching, for example keypoint extraction andGeneralized Hough Transform, tend to use a “voting” procedure, wherecertain matching points on a shape are found, and for each possibleposition and orientation of an object, a “vote” is taken. This hasseveral advantages: it is robust to occlusions/missing parts of an image(by tolerating a certain number of missing votes); it is robust to smallrotations and perspective distortions; and it can be made tolerant tolarger rotations and perspective distortions. If the objectcorresponding to the template is not present in the image to which thevoting procedure is applied, there may be a number of background “votes”from matches in portions of the image that do not relate to the objectand so a minimum number of votes threshold is used to detect that anobject is present in an image. However, for the unfurled images, the useof standard keypoint extraction features did not prove reliable.

One approach is to combine a voting method with template matching bybreaking up a template into smaller tiles and FIG. 19 shows a largetemplate 1910, in this case of the label RoActemra® along with threesmall sub tiles thereof 1912, 1914, 1916. Although only three tiles areshown in FIG. 19 all, or substantially all, of the chosen template wouldpreferably be covered by the tiles so that all, or substantially all, ofthe information contained within the template is also contained withinthe set of tiles. Template matching is then performed for tile in turn.Because each tile is small, the computational cost of processing all thetiles is similar to the cost of a full template match. Although anygiven tile may be found to be a best match at a location in the unfurledimage that does not lie on the object represented by the template, it isunlikely that a plurality of matches that are close to the correctspatial arrangement will occur by chance.

Following template matching for each tile, each tile votes on where itthinks the “best” location is (mapped back to the centre of the originaltemplate) using the below-described approach which creates a votingimage V_(t) having the same dimensionality as the binarised image.

The voting algorithm takes as its inputs an image to test, I, and a setof binarised templates. As a preprocessing step, for each template, T,that template is divided into N tiles. In the follow description thesubscript t is used to denote values that relate specifically to thet^(th) tile. For each tile, its location relative to the top left handcorner of the full template is stored. f is the number of pixels betweenthe left hand edge of the full template and the left hand edge of thetile. Likewise g is the number of pixels between the top edge of thefull template and the top edge of the tile.

At run-time:

1. For each template

-   -   a. For each tile in the current template i. Compute:

${{M_{t}\left( {{x - f},{y - g}} \right)} = {\sum\limits_{x^{\prime},y^{\prime}}{{T_{t}^{\prime}\left( {x^{\prime},y^{\prime}} \right)}{I^{\prime}\left( {{x + x^{\prime}},{y + y^{\prime}}} \right)}}}}{Where}{{T_{t}^{\prime}\left( {x^{\prime},y^{\prime}} \right)} = {{T_{t}\left( {x^{\prime},y^{\prime}} \right)} - {\frac{1}{wh}{\sum\limits_{x^{''},y^{''}}{T_{t}\left( {x^{''},y^{''}} \right)}}}}}{and}{{I^{\prime}\left( {x^{\prime},y^{\prime}} \right)} = {{I\left( {x^{\prime},y^{\prime}} \right)} - {\frac{1}{wh}{\sum\limits_{x^{''},y^{''}}{I\left( {{x^{\prime} + x^{''}},{x^{\prime} + y^{''}}} \right)}}}}}$where the summations are taken over the dimensions of the template andwh is a weighting factor and the response image M_(t) is offset by anamount (f,g) to account for the relative position of the tile within thetemplate.

The scores from the individual templates are combined in the followingway. First the sub-template matching scores are converted to votes:

$V_{t} = \left\{ \begin{matrix}{{0{if}M_{t}} < T_{t}} \\{{w_{t}{if}M_{t}} \geq T_{t}}\end{matrix} \right.$

Then the individual votes are summed:V=Σ _(t=1) ^(N) ^(T) V _(t)

Finally this is relaxed or blurred, in this case by convolving V with asquare window. It is this final step that provides some scale and skewrobustness by effectively allowing the different sub-templates to bemoved slightly relative to one another:

${\mathcal{S}\left( {x,y} \right)} = {\sum\limits_{u = {- C}}^{C}{\sum\limits_{v = {- C}}^{C}{V\left( {{x - u},{y - v}} \right)}}}$

The final score for the template is taken as the maximum value in theimage S(x,y) and the template is deemed to be located at that point.

As one example, for getting a match between a template and the label ofa syringe, the classification involves: splitting a 25×150 template into30 patches of 5×25 and then, for each patch: computing the sum ofsquared differences at each possible position in the label so as toproduce a score; determining the maximum score in a label, and setting athreshold at 90% of the determined maximum score; marking the positionsin the label where the score is above the threshold with a 1 (and a 0otherwise) and counting those positions (denote by N); giving eachposition marked a 1 a computed value of 1/N; summing, for each patch,the computed values of each of the positions in the label; andidentifying the position having the highest summed value.

FIG. 20 shows an example unfurled image 2010, alongside a response for astandard template matching algorithm 2012 using a “COPAXONE” templateand S(x,y) 2014 for the same template. It can be seen that S(x,y) 2014shows a strong response at the correct location of the COPAXONEtemplate.

FIG. 21 shows another example unfurled image 2110, alongside a responsefor a standard template matching algorithm 2112 using a “COPAXONE”template and S(x,y) 2114 for the same template. It can be seen that theresponse for the standard template matching algorithm 2112 is a poormatch to the template whereas S(x,y) 2114 shows a strong response at thecorrect location of the COPAXONE template. For the example of FIG. 21 ,there is some perspective distortion remaining in the unfurled image2110, but in contrast to the standard template matching, the votingalgorithm is tolerant to that. The peak is correspondingly less sharp,but still present.

Although the template matching approach described above works well formatching text, it is not so effective for cases where the template is ablock of colour. As an example, for the drug Saizen blocks of colourprovide valuable information about the drug-containing vessel as thelabel is yellow and has a yellow rectangle for a 20 mg cartridge and isred and has a red rectangle for a 12 mg cartridge. For such cases,instead of the template being chosen to represent writing on the label,it may instead be chosen to be a block of a given colour and size andtemplate matching is then performed using sum of squared differences tocalculate the quality of the template match. For colour block templateshue based binarisation was used. The colour binarisation processes wereconfigured to accept a wide range of hues around the expected hue of thecolour block. This was so that the identification was robust to a rangeof lighting conditions and would also mean that process would be robustto printing variations. This works well since blocks of uniform colourare quite robust to the perspective distortions that necessitate thevoting based template matching scheme described above. Note that thistemplate matching algorithm is applied to the colour image and not to abinarised version. For colour block templates, hue-based binarisationcan be appropriate and the binarisation processes configured to accept awide range of hues around the expected hue of the colour block. Thismakes the identification robust to a range of lighting conditions andalso provides robustness in relation to printing variations.

Each label to be classified may have multiple templates associated withit. Examples of the types of template that a single label may haveinclude: a template containing the name of the drug, a templatecontaining text specifying the dose, a template of a block of colourthat helps to identify the drug type or dose, a template containingfeatures that are not expected to be present. Templates containingfeatures that are not expected to be present can help make theclassifier more robust in cases where there are known to be similarlabels as looking for features that should not be present on a similarlabels can help prevent the classifier from incorrectly accepting suchlabels.

The unfurled image to be classified will produce a template matchingscore for each of the templates that are evaluated against it. Thesescores are then converted into a classification result. This is done byapplying a threshold to each of the features and accepting the label ifthe template matching scores are above the threshold for each of therequired templates and rejecting the label the label if the templatematching scores are below the threshold for templates that should not bepresent as illustrated in FIG. 22 .

As one possibility, in order to reduce computational complexity only asub-region of the unfurled image may be searched when performingtemplate matching. In particular, while the vertical position of thedrug name could be anywhere, the horizontal position will only vary asmall amount so the search may be constrained to occur within certainhorizontal bounds—for example 10% of the image width around the expectedhorizontal position of the object represented by the drug name.

As one possibility, in order to reduce computational complexity, theresolution of the unfurled (or even imaging device) image(s) could bereduced. Although the results illustrated in FIGS. 20 and 21 wereachieved for full resolution images, the approaches described hereincould also be performed following an additional step of reducing theresolution of one or more of the imaging device image(s) and theunfurled image.

As one possibility, in order to reduce computational complexity, acascaded approach could be employed wherein only a subset of the tiles(for example 3) are used during a first stage of the template matchingso as to enable a quick initial estimation of the location of thetemplate in the unfurled image before constrained template matching isperformed with others of the tiles wherein the constrained templatematching limits the distance from the initial estimate that theoptimization is performed for the others of the tiles.

A large amount of the computational cost of the template matchingapproach comes from the fact that a large part of the image needs to besearched to find the part that contains the drug name. This comes fromthe fact that the drug-containing vessel may be in different rotationalorientations. As one possibility, as the amount of dark pixels v lightpixels in each row will vary with what is present in that part of theimage, a metric of the dark pixels v light pixels in each row could becalculated and registered to the rotational orientation of the labelthat best matches the metric. The above described pattern matching wouldthen be performed but, as the potential location drug name would beknown to a much higher degree, a constrained template matching would beperformed based on the registration thereby allowing the approach toperform whilst searching a much reduced portion the unfurled image.

The approaches described herein have been found to take in the range of10 ms to 15 s and may take 200 ms to determine whether the label is of agiven type and are estimated to require in the range of 0.05 mAh to 0.5mAh and may take 0.06 mAh of processor and acquisition energy per labelidentification. When an apparatus arranged to perform the templatematching approaches described herein needs to be able to recognize a newlabel, a new template can simply be supplied to the apparatus therebyenabling adaptation of the apparatus to recognize new labels without theneed for a fundamental changing of the apparatus' processing code.

As one possibility, once an autoinjector has identified that it iscarrying a particular drug-containing vessel, it may then proceed topermit injection from that drug-containing vessel. In cases where anautoinjector identifies that it is carrying a drug-containing vesselother than one that it is expecting to be carrying, it may issue avisual or audible alarm and/or disable its injection capabilities.

As one possibility, a light source is provided within the autoinjectorand used to illuminate the curved surface of the cylindrical portion tohelp mitigate image processing issues caused by inconsistentillumination.

As one possibility, instead or, or as well as, using the above describedtemplate matching approach, a neural network based classificationapproach may be employed to identify the drug and/or details about thedrug-containing vessel from the label image. An n×m pixel RGB image hasa base dimensionality of 3 nm; so for a 2500×1900 image, this wouldresult in an input vector of dimension of approximately 14 million.Accordingly the basic approach is to cut the unfurled image up into aset of smaller o×p image tiles (for example 15), compute arepresentative “feature” metric on each of these tiles, and then usethis feature vector as input into the neural network. Preferably, themetric which will capture, with as few numbers as possible, the salientfeatures of the tile which lend themselves to some degree of separation.For the metric on each tile, a guiding heuristic was to use one whichwould somehow capture both high frequency and low frequencycharacteristics. The metric chosen was a concatenation of data energyand the mean value of each of the three colour channels, which gives ametric dimension of 4. With this metric the size of the input vectorbecomes 4rs, and using a tiling dimension of 5×3 gives and input vectordimension of 60. This is close to a six order of magnitude reduction inthe input vector size. The network used was a 60 input, one hiddenlayer, single output fully connected network. The size of the hiddenlayer is 20. One feature of the network was the use of a Gaussianactivation function for the perceptrons in the hidden layer. The networkwas then trained on a set of training images for which the druginformation is known.

Where mention is made herein to template matching, it is contemplatedthat, as one possibility, the medical device would be able to performthe template matching without the need to interrogate any externaldatabase. In such circumstances, the medical device may have stored inits memory one or more templates corresponding to candidate informationabout the vessel and/or the drug. Furthermore, although it may bebeneficial for the template matching algorithm to learn from the matchesthat it makes and/or does not make, the template matching approach neednot have any such learning capability.

It is contemplated that the imaging device or devices employed with theapproaches described herein could be one or more cameras and so any useherein of the term “imaging device” could be replaced with the term“camera”.

As one possibility, instead of the above described network architecturehaving only one output, which means it is unable to reject any images asbeing invalid. A multi-output network could instead be employed. Onesuch topology could have eight (or more of less) outputs (one for eachlabel category), with the expectation that for a given label there wouldbe a high value on that label's output with all other outputs being low.Any output pattern which deviated from this pattern would then beinterpreted as a rejection.

Although the approaches described herein may be employed with andimplemented in any medical device, as one possibility, they may beimplemented in a handheld medical device, such as an autoinjector. Asanother possibility, they may be implemented in a device that is nothandheld such as a sharps bin.

In an approach described herein, information about a drug-containingvessel is determined by capturing image data of the curved surface of acylindrical portion of a drug-containing vessel. The image data isunfurled from around the curved surface, binarised, and a templatematching algorithm employed to determine that the label informationcomprises candidate information about the vessel and/or the drug.

As one possibility any of the approaches described herein may beemployed in another approach wherein the label information is read fromthe unfurled image without using template matching—for example usingtext, bar code, QRS, and/or recognition approaches.

Methods described herein can be computer-implemented so as to becausable by the operation of a processor executing instructions. Theapproaches described herein may be embodied in any appropriate formincluding hardware, firmware, and/or software, for example on a computerreadable medium, which may be a non-transitory computer readable medium.The computer readable medium carrying computer readable instructionsarranged for execution upon a processor so as to make the processorcarry out any or all of the methods described herein—thereby making suchmethods computer implemented.

The term computer readable medium as used herein refers to any mediumthat stores data and/or instructions for causing a processor to operatein a specific manner. Such a storage medium may comprise non-volatilemedia and/or volatile media. Non-volatile media may include, forexample, optical or magnetic disks. Volatile media may include dynamicmemory. Exemplary forms of storage medium include, a floppy disk, aflexible disk, a hard disk, a solid state drive, a magnetic tape, anyother magnetic data storage medium, a CD-ROM, any other optical datastorage medium, any physical medium with one or more patterns of holesor protrusions, a RAM, a PROM, an EPROM, a FLASH-EPROM, NVRAM, and anyother memory chip or cartridge.

The invention claimed is:
 1. A computer-implemented method ofdetermining information about a drug-containing vessel and/or the drugitself, wherein the vessel is located within a medical device and has acylindrical portion bearing label information about the vessel and/orthe drug, the method comprising: capturing, using one or more imagingdevices contained within the medical device, image data of the curvedsurface of the cylindrical portion; creating a two-dimensional unfurledimage from the image data by identifying where points in the unfurledimage would map to on the curved surface if the unfurled image wasfurled about the curved surface; binarising the unfurled image; applyinga template matching algorithm to the binarised image to determine thepresence in the binarised image of one or more templates correspondingto candidate information about the vessel and/or the drug; and based onthe determination of the presence in the binarised image of the one ormore templates, determining that the label information comprises thecandidate information, wherein the template matching algorithm isarranged to evaluate each of the templates against the binarised imageby evaluating a plurality of tile portions of the respective templateagainst the binarised image, and wherein, for each template, a votingimage having the same dimensionality as the binarised image is createdand its pixels populated based on votes from each of the plurality oftile portions of that template as to whether or not the evaluation ofthe respective tile portion was above or below a predetermined thresholdfor that pixel, and the maximum pixel value of each voting image is usedto determine the presence in the binarised image of that template, andwherein each voting image is blurred prior to determining its maximumpixel value.
 2. The method of claim 1, further comprising, subsequent tocreating the unfurled image and prior to binarising the unfurled image,performing a patching operation on the unfurled image to patch togetherportions of the image data that were captured by different ones of theone or more imaging devices.
 3. The method of claim 1, wherein thecreation of the two-dimensional unfurled image is performed so that theimage data that was captured by each imaging device is only unfurled inrespect of the portion of the curved surface of the cylindrical portionthat was directly observable by that imaging device at the time ofcapturing.
 4. The method of claim 1, wherein the binarising comprises:converting the unfurled image into a greyscale image; and creating thebinarised image such that each pixel has a first value if the greyscalevalue of that pixel of the greyscale image data is less than a localmean intensity threshold and otherwise has a second value.
 5. The methodof claim 1, wherein the binarising comprises: converting the unfurledimage into an Hue-Saturation-Value (HSV) image; and creating thebinarised image such that each pixel has a first value if each of theValue-Hue-Saturation values of that pixel lie between respectiveHue-Saturation-Value upper and lower thresholds and otherwise has asecond value.
 6. The method of claim 1, further comprising: applying atemplate matching algorithm to the unfurled image to determine thepresence in the binarised image of one or more coloured templatescorresponding to coloured candidate information about the vessel and/orthe drug; and based on the determination of the presence in the unfurledimage of the one or more coloured templates, determining that the labelinformation comprises the coloured candidate information.
 7. The methodof claim 1, wherein the blurring of the voting image is achieved byconvolving the voting image with a square window.
 8. The method of claim1, wherein the vessel is a syringe or cartridge.
 9. The method of claim1, further comprising using a light source within the medical device toilluminate the curved surface of the cylindrical portion.
 10. A medicaldevice arranged to perform the method of claim 1, wherein the one ormore imaging devices are cameras.
 11. A non-transitory computer readablemedium comprising machine readable instructions arranged, when executedby one or more processors, to cause the one or more processors to carryout a computer-implemented method of determining information about adrug-containing vessel and/or the drug itself, wherein the vessel islocated within a medical device and has a cylindrical portion bearinglabel information about the vessel and/or the drug, the methodcomprising: capturing, using one or more imaging devices containedwithin the medical device, image data of the curved surface of thecylindrical portion; creating a two-dimensional unfurled image from theimage data by identifying where points in the unfurled image would mapto on the curved surface if the unfurled image was furled about thecurved surface; binarising the unfurled image; applying a templatematching algorithm to the binarised image to determine the presence inthe binarised image of one or more templates corresponding to candidateinformation about the vessel and/or the drug; and based on thedetermination of the presence in the binarised image of the one or moretemplates, determining that the label information comprises thecandidate information, wherein the template matching algorithm isarranged to evaluate each of the templates against the binarised imageby evaluating a plurality of tile portions of the respective templateagainst the binarised image, and wherein, for each template, a votingimage having the same dimensionality as the binarised image is createdand its pixels populated based on votes from each of the plurality oftile portions of that template as to whether or not the evaluation ofthe respective tile portion was above or below a predetermined thresholdfor that pixel, and the maximum pixel value of each voting image is usedto determine the presence in the binarised image of that template, andwherein each voting image is blurred prior to determining its maximumpixel value.